Distributed connectionist networks have no facility for incremental learning, but they have the advantage of being able to generalize. In contrast, winner-take-all networks are suitable for incremental learning but lack the ability to generalize. In this paper, we use an abstract model to assess the trade-off between incremental learning and generalization abilities, and we propose a new model to solve this dilemma. To formulate and analyze the trade-off, we have defined a network that emulates both the connectionist network and the winner-take-all network. It does this through a single parameter that specifies the range of lateral inhibition and varies continuously and gradually between the distributed and winner-take-all networks. By using structured mutual inhibition instead of simple lateral inhibition, the network is able to balance the ability to learn incrementally with the ability to generalize. We also analyze the behavior of the proposed mechanisms using Formal Concept Analysis, which reveals that the network can form concepts that are defined by firing patterns in network subgroups. Based on these results, we propose that longstanding perspectives on this underlying dilemma in connectionism should be shifted and that the tradeoff problem needs to be solved through a new conceptualization.