Single-molecule data often come in the form of stochastic time trajectories. A key question is how to extract an underlying kinetic model from the data. A traditional approach is to assume some discrete state model, that is, a model topology, and to assume that transitions between states are Markovian. The transition rates are then selected according to which ones best fit the data. However, in experiments, each apparent state can be a broad ensemble of states or can be hiding multiple interconverting states. Here, we describe a more general approach called the non-Markov memory kernel (NMMK) method. The idea is to begin with a very broad class of non-Markov models and to let the data directly select for the best possible model. To do so, we adapt an image reconstruction approach that is grounded in maximum entropy. The NMMK method is not limited to discrete state models for the data; it yields a unique model given the data, it gives error bars for the model, and it does not assume Markov dynamics. Furthermore, NMMK is less wasteful of data by letting the entire data set determine the model. When the data warrants, the NMMK gives a memory kernel that is Markovian. We highlight, by numerical example, how conformational memory extracted using this method can be translated into useful mechanistic insight.