Human hippocampus represents space and time during retrieval of real-world memories

Significance The rodent hippocampus contains neurons that code for space on the scale of meters, a discovery that was recently awarded a Nobel Prize. However, it remains unclear whether humans harness similar representations for memory at the scale of their lives. Our results reveal that the human hippocampus represents the spatial and temporal location of memories for real-life events at scales of up to 30 km and a month of time. They further suggest that real-world representations of space and time are more intimately entwined in the hippocampus than previously thought, validating the relevance of decades of rodent studies for human memory and providing a potential mechanism for how we weave the episodic memories of our lives. Memory stretches over a lifetime. In controlled laboratory settings, the hippocampus and other medial temporal lobe brain structures have been shown to represent space and time on the scale of meters and seconds. It remains unclear whether the hippocampus also represents space and time over the longer scales necessary for human episodic memory. We recorded neural activity while participants relived their own experiences, cued by photographs taken with a custom lifelogging device. We found that the left anterior hippocampus represents space and time for a month of remembered events occurring over distances of up to 30 km. Although previous studies have identified similar drifts in representational similarity across space or time over the relatively brief time scales (seconds to minutes) that characterize individual episodic memories, our results provide compelling evidence that a similar pattern of spatiotemporal organization also exists for organizing distinct memories that are distant in space and time. These results further support the emerging view that the anterior, as opposed to posterior, hippocampus integrates distinct experiences, thereby providing a scaffold for encoding and retrieval of autobiographical memories on the scale of our lives.

Memory stretches over a lifetime. In controlled laboratory settings, the hippocampus and other medial temporal lobe brain structures have been shown to represent space and time on the scale of meters and seconds. It remains unclear whether the hippocampus also represents space and time over the longer scales necessary for human episodic memory. We recorded neural activity while participants relived their own experiences, cued by photographs taken with a custom lifelogging device. We found that the left anterior hippocampus represents space and time for a month of remembered events occurring over distances of up to 30 km. Although previous studies have identified similar drifts in representational similarity across space or time over the relatively brief time scales (seconds to minutes) that characterize individual episodic memories, our results provide compelling evidence that a similar pattern of spatiotemporal organization also exists for organizing distinct memories that are distant in space and time. These results further support the emerging view that the anterior, as opposed to posterior, hippocampus integrates distinct experiences, thereby providing a scaffold for encoding and retrieval of autobiographical memories on the scale of our lives. hippocampus | representational similarity analysis | lifelogging | episodic memory T he hippocampus plays a critical role in remembering the events of our lives (1). Direct evidence from single-neuron recordings in rats indicates that cells in the hippocampus fire in specific spatial locations (2)(3)(4)(5)(6) or at specific times during a temporal delay (7,8). Single-neuron and functional MRI (fMRI) studies in individuals navigating virtual environments have confirmed that cells coding for spatial location are also present in the human hippocampus (9)(10)(11). Similarly, place-responsive cell activity recorded in the hippocampus of patients with epilepsy during navigation of a virtual town was shown to reinstate during episodic memory retrieval of the previous virtual navigation (12). Together, these studies provide evidence that the same neurons in the medial temporal lobe (MTL) that are active during an experience also help represent the memory for that experience. These results, however, are limited to simple events in laboratory settings that occur on the scale of minutes and meters, thereby leaving unanswered whether we harness similar mechanisms in more natural settings and over larger temporal and spatial scales.
Recent studies have used more naturalistic designs with incidentally acquired memories recorded via lifelogging devices that automatically capture photographs from the participants' lives (13,14). The typical finding is increased hippocampal activation when participants view images from their cameras as opposed to images from other participants' cameras (15)(16)(17), and this activation decays over the course of months (14). Still, there is no evidence to date that the hippocampus or other MTL structures actually represent space or time of autobiographical experiences. We addressed this question by having participants relive their own real-life experiences in the fMRI scanner. We then used multivariate pattern analysis (18) to identify regions of the MTL that represent space and time of these remembered experiences. If a brain region represented either space or time of personal experiences, the distances between neural activity patterns would correlate with the spatial or temporal proximity of the original experiences.

Results
Nine participants wore smart phones ( Fig. S1) with our custom lifelogging software that captured images along with their global positioning system (GPS) coordinates and timestamps (19). We collected an average of 5,414 ± 578 SEM images per subject, primarily from the Columbus, Ohio metropolitan area (Fig. 1A). We selected 120 images from each participant's data to present to the participant in the fMRI scanner (Methods). Participants were instructed to "try to remember the event depicted in the picture, and try to mentally relive your experience" while viewing each image for 8 s. Focusing only on the memories subjects recalled (63.4% ± 4.7 SEM per subject; Fig. S2), we calculated the temporal and geospatial distance for each pair of events and the corresponding neural distances for four bilateral MTL regions and one bilateral primary visual cortex region of interest (ROI) (Fig. 1 B-D). We excluded image pairs taken close together in space or time to prevent directly overlapping events (Methods and Fig. S3). After applying these selection criteria, we had an average of 76.1 ± 5.7 SEM images and 1,995.7 ± 261.5 SEM image pairs per subject.
We applied representational similarity analysis to identify whether regions of the MTL reflect the spatial and temporal dimensions of episodic memories (18). To quantify the role of multiple factors in a single model, we implemented a general linear model (GLM) within each subject as the basic test of our

Significance
The rodent hippocampus contains neurons that code for space on the scale of meters, a discovery that was recently awarded a Nobel Prize. However, it remains unclear whether humans harness similar representations for memory at the scale of their lives. Our results reveal that the human hippocampus represents the spatial and temporal location of memories for reallife events at scales of up to 30 km and a month of time. They further suggest that real-world representations of space and time are more intimately entwined in the hippocampus than previously thought, validating the relevance of decades of rodent studies for human memory and providing a potential mechanism for how we weave the episodic memories of our lives. This article is a PNAS Direct Submission.
Data deposition: The data reported in this paper are available via the corresponding author's institutional website. 1 To whom correspondence should be addressed. Email: sederberg.1@osu.edu.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10. 1073/pnas.1507104112/-/DCSupplemental. representational similarity analysis. Thus, in the eight MTL and two primary visual regions, we tested the hypothesis that differences in the patterns of neural activity while remembering events are correlated with the spatial and temporal distance separating those events. Because the temporal distance between recollection of events in the scanner could confound temporal distance between the events themselves, we included the time between image presentations as a nuisance regressor. Based on work implying scale-free neural representation of time and space (20), we calculated the log of the spatial and temporal distances between events and the times between presentations in the scanner. The pairwise nature of the distances in our analysis violates the assumption of independent samples, so we used nonparametric permutation to test for significance (21). We permuted the neural distances with respect to the behavioral distances 10,000 times to generate a null dataset from which we calculated nonparametric P values, which we then Bonferroni-corrected based on the 10 regions in our analysis. We found that spatial distance (corrected P = 0.021), temporal distance (corrected P = 0.010), and their interaction (corrected P = 0.038) were significantly correlated with neural distance in the left anterior hippocampus ( Fig. 2 and Table S1). No significant correlations were found for the nonrecalled events. These results indicate that the left anterior hippocampus represents space and time for memories of real-life experiences at larger scales than ever before tested.
Although we attempted to remove any pairs of images that were from the same (or proximal) episode, it is still possible that some aspect of the images themselves could account for the systematic changes in neural activity in the left anterior hippocampus. As a control, we compared the magnitude of the spatial and temporal effects in the left anterior hippocampus with equivalently sized ROIs in the right and left primary visual cortex. The spatial and temporal distances were more strongly associated with neural distance in the left anterior hippocampus than in either the right anterior hippocampus or the right or left primary visual cortex (Fig. 3, Fig. S4, and SI Results), suggesting that our results are not driven simply by the image properties alone. We also ruled out the possibility that recency, proximity, or contiguity effects were responsible for the result we observed (Figs. S5-S7 and SI Results).

Discussion
We have shown that the left anterior hippocampus represented the spatial and temporal dimensions of remembered experiences for distances between 100 m and 30 km and for times between 15 h and 1 mo. Although previous studies have identified similar drifts in representational similarity across space or time over the relatively brief time scales (seconds to minutes) that characterize individual episodic memories (22)(23)(24), our results provide compelling evidence that a similar pattern of spatiotemporal organization also exists for organizing distinct memories that are distant in space and time. We did not observe a significant effect of recency or proximity of the images on the likelihood that the associated episodes would be recalled. This finding indicates that the relationship we observed between activity patterns in the left anterior hippocampus and space and time is unlikely to be caused by simple recency or proximity effects. We did find that spatial or temporal contiguity between a recalled image and a subsequently presented image increased the likelihood that the subsequently presented item would be recalled, but only when we included pairs of images from the same event in our analysis. When we excluded pairs of images from the same event, or when we applied the limits on spatial and temporal distances between pairs of images used in our main analysis, we did not find that spatial or temporal contiguity effects were significant. Taken together with the absence of a significant recency or proximity effect, the most likely explanation for the observed relationship between patterns of neural activity in the left anterior hippocampus and spatial and temporal distance is that, in fact, information about spatial and temporal distances is represented in the left anterior hippocampus.
There are some important limitations in the work that we have presented here. By chance, our sample is composed entirely of female participants; therefore, we cannot say with certainty that these results will be generalizable to representation of space and time in males. Another limitation of this work is the spatial resolution of our fMRI, which prevents us from examining the roles of distinct hippocampal subfields. We are also unable to analyze the representation of space and time within events because relatively few images from the same event were presented to each subject. A future study with a more balanced sampling of within-and between-event images may be able to distinguish representation of space and time at these different scales.
Prior fMRI results examining judgment of landmark locations demonstrated that left anterior hippocampal activity is parametrically related to the real-world distance between the currently presented landmark and one that was just seen on the previous trial, providing indirect evidence that the left anterior hippocampus represents spatial information (25). However, both Morgan et al. (25) and a subsequent study (26) were unable to detect metric-based information about spatial distances in the anterior hippocampus using multivoxel pattern analysis. These studies did not use images that represented participants' previous experiences, so they were unable to include time or the interaction of time and space in their model. We found that neural activity in the left anterior hippocampus correlated with space, time, and their interaction, indicating that mental representations of space and time are perhaps not as distinct as we generally conceive them to be.
Fueling a long-standing debate over the lateralization of episodic memory processes (27), recent work in mice showed that the left, but not the right, hippocampus was essential for associative spatial long-term memory (28), and, as mentioned above, recent work demonstrated that processing of spatial information may be left-lateralized in the human hippocampus as well (25). To test whether our data followed a similar pattern, we compared the left and right anterior hippocampi directly. We found that the left anterior hippocampus represents space and time significantly better than the right anterior hippocampus (Fig. 3). This finding extends the emerging theory of hippocampal lateralization with evidence that the memory representations of reallife events are left-lateralized in the human hippocampus.
In rats, the dorsal hippocampus (analogous to the human posterior hippocampus) represents shorter temporal and spatial scales, whereas the ventral hippocampus (analogous to the human anterior hippocampus) represents longer scales (29)(30)(31). Our finding of long spatial scale representation in the human anterior hippocampus, combined with the established involvement of the human posterior hippocampus in memory for shorter spatial scales (32,33), supports the hypothesis that humans have a gradient of spatial scale along the anteroposterior axis of the hippocampus. Our results suggest that the the maximum scale of spatial representation in the human hippocampus is at least the 30 km we observed here, which is considerably larger than the maximum of about 10 m observed in rats (29). It is possible, however, that the maximal scale of representation is not limited by biology, but by experience. Humans can regularly travel tens, if not hundreds, of kilometers in short periods of time, and so it is in some ways unsurprising that the maximal scale of representation in humans is significantly larger than in rodents raised in a laboratory, which travel, at most, tens of meters in a day.
Taken together, our results point to a critically important role of the left anterior hippocampus in representing our past experiences on the scale of our lives. These large spatial scale representations of space and time give structure to our memories, which may allow us to search our past (34) or imagine the future (35) efficiently, and to make associations and generalizations across nonoverlapping events (30,31). Based on our current understanding of hippocampal representations, place cells provide spatial relationships (5) and time cells provide temporal relationships (7), yet there is also evidence from recordings in the CA1 subregion of the hippocampus that place codes change over hours and days Fig. 3. Comparison of space and time representations across the brain region. Space and time are significantly more strongly related to neural distance in the left anterior hippocampus (LAH) than in the right anterior hippocampus (RAH) or the right or left primary visual cortex (RV1 and LV1, respectively). *P < 0.05; **P < 0.01. (36) to weeks (37). The conjunctive spatiotemporal representations we observed suggest space and time are more intimately related in the hippocampus than thought before, providing the thread for weaving the episodic memories of our lives (38).

Methods
Device and Software. Each participant carried an Android-based smartphone in a pouch attached to a neck strap as shown in Fig. S1 from morning until evening. The smartphone was equipped with a custom lifelogging application that acquired image, time, audio (obfuscated), GPS, accelerometer, and orientation information throughout the day. The participants had control over what data they wanted to share with the experimenters. They were instructed on how to delete data from the phone. They were also allowed to turn the application off or to place a flap over the camera lens at any time during the data collection period when they felt the need for privacy.
The lifelogging application was written by our programmers using Java (Oracle Corporation) to run in the background as a service. Data acquisition times can be fixed or variable, and they can be determined by a movementbased trigger to preserve battery resources when the user is not very active. When the smartphone detects WiFi and is connected to a charger, it sends the collected and stored data automatically to a remote server. This transmission usually happened once per day at the end of the day because users charged the phone overnight. The data were sent in batch mode via SFTP (Secure File Transfer Protocol) for added security and remained inaccessible to other users in the system.
Participants. Participants were recruited using advertisements placed on notice boards in multiple buildings on the main campus of The Ohio State University. To join the study, potential participants had to be willing to participate in the lifelogging data collection and to be willing and able to undergo an MRI scan. Potential participants with contraindications for receiving an MRI scan were excluded. They were compensated at the rate of $10 per day for wearing the smartphone to collect data and at the rate of $15 per hour for the fMRI session. We recruited 10 participants (aged 19-26 y, mean age = 21.4 y; nine female), nine of whom wore the smartphone for ∼1 mo. The tenth participant wore the smartphone for 2 wk. One participant (male) did not complete the fMRI session due to discomfort in the scanner; therefore, we did not include the data for that participant in any of our analyses. Our study has a similar number of participants as other fMRI studies using lifelogging devices, such as the study of Cabeza et al. (15), which had 13 participants and only collected lifelogging data for 10 d, and the study of Milton et al. (14), which had 10 participants and only collected lifelogging data for 2 d, although there was a 5-mo follow-up in their study.
Ethics Statement. The research protocol was reviewed and approved by the Institutional Review Board at The Ohio State University. Written informed consent was obtained from all participants, once before the lifelogging data collection phase and once before the fMRI session. Consent was obtained from one participant to use images and locations from that individual's lifelogging data for Fig. 1.
Other Behavioral Tasks. At the end of each day, the smartphone was connected to a power outlet to be charged overnight. When connected to the charger and to the Internet, the smartphone automatically uploads data to our server. The images were viewed on a web interface, whose link was uniquely generated for each participant and provided to the participant before data collection. Participants accessed their images on the web interface via a unique link, where they segmented their stream of images into distinct episodes and tagged each episode with a set of tags chosen from a drop-down menu (39). For each episode, they also provided a brief title and description.
After they collected data for 2 wk, participants came into the laboratory on the Thursday of the third week for a weekly discrimination test. Each participant's test was based on images drawn from her own lifelogs. The weekly discrimination task was described to the participants, and they were told that only images from the weekdays of the preceding 2 wk would be presented on the computer screen. The pictures remained on the screen while they made the weekly judgment, and they could use as much time as they needed to respond. The results of this behavioral task will be analyzed and presented separately.
GPS Data. The GPS receivers in our lifelogging smartphones were unable to record GPS coordinates when participants were inside some buildings. In these cases, we attempted to determine the coordinates based on the participant's description of the location and by traveling to the approximate location where the image was taken and searching the area until the scene captured in the image could be identified. We determined the approximate area to search based on GPS data acquired before or after the image with missing GPS data. Once the location had been determined, GPS coordinates were obtained from Google Maps.
Stimuli Selection. We selected 120 images from each subject's lifelogging data to present to the subject in the scanner. First, we excluded pictures of floors/ ceilings/walls, blurry images, and images with inadequate exposure. Then, we selected images that appeared to have enough detail that they could act as cues for distinct episodes. From this subset of images, we selected images representing events that spanned the entire period each participant wore the lifelogging device, with as uniform sampling of events as possible. We did not take spatial location into account when selecting the images.
fMRI Experiment. In the scanner, subjects were instructed that they would be viewing images from the experience sampling experiment they recently completed and told that each image would be displayed for 8 s. Subjects were asked to ". . . try to remember the event depicted in the picture, and try to relive your experience mentally." After the remembrance period for each event, subjects were asked if they remembered the event ("yes" or "no") and how vividly they recalled the event ("lots of detail" or "very little detail"). Participants were given 2.5 s to respond to each of those questions using a button box held in their right hand. The images were presented in random order, and the task was split into eight runs with 15 images in each run. With each image presented for 8 s and each question for presented 2.5 s with a 0.5-s interstimulus interval, each trial took a total of 14 s. The intertrial interval was jittered uniformly between 4 and 10 s, allowing for a true event-related design.
fMRI Processing. fMRI processing was carried out using Analysis of Functional NeuroImages (AFNI) (40) and Functional Magnetic Resonance Imaging of the Brain (FMRIB) Software Library (FSL) (41). The T1-weighted anatomical image was intensity-normalized, skull-stripped, and warped to a 2.5-mm MNI-152 template using 3dQwarp. We selected a 2.5-mm template to match the resolution of the functional scans. For the functional scans, we dropped the first two TRs of each run, then removed spikes with 3ddespike and temporally shifted all of the slices in each volume to the start of the TR using 3dTshift with Fourier interpolation. We then warped the functional scans to template space, blurred them to 4-mm FWHM using 3dBlurtoFWHM, and scaled the voxel values to a mean of 100 (maximum of 200) for each run. At this point, we performed independent component analysis of each functional run with FSL's MELODIC. Components were visually inspected to identify noise components following published guidelines (42). Noise components were regressed out of the functional runs using FSL's fsl_regfilt command. We then ran a regression with restricted maximum likelihood estimation of temporal autocorrelation structure on the filtered functional runs using 3dDeconvolve and 3dREMLfit to generate single-trial betas for each reminiscence trial and to regress out the effects of the mean and derivative of motion terms, as well as cerebrospinal fluid signal. The regressor for each image presentation was an 8-s block convolved with a hemodynamic response function. The neural activity of the question prompts were accounted for with a 2.5-s block convoluted with a hemodynamic response function. We modeled response processing and motor activity related to the button push with a set of nine tent functions over the 16 s after the question response. Including these tent functions in our model allowed us to estimate the motor response robustly for each subject so that the signal from the motor responses did not contaminate the single-trial beta fit for each reminiscence period. Lastly, we regressed out local white matter signal with 3dAnaticor. Researchers were not blinded during preprocessing or subsequent analyses.
Previous work has shown that the MTL represents space and time for short distances, so we focused our analysis on regions in the MTL using ROI analysis because the shape of the MTL structures is often problematic for multiple comparison correction procedures. We identified four ROIs from each hemisphere in the MTL: anterior hippocampus, middle hippocampus, posterior hippocampus, and parahippocampal cortex. The parahippocampal ROIs were taken from the Harvard-Oxford subcortical atlas (43), down-sampled to 2.5 mm. The parahippocampal ROI in this atlas includes the parahippocampal, perirhinal, and entorhinal cortices. The hippocampal ROIs were derived by resampling the Harvard-Oxford subcortical atlas to 2.5 mm, extracting the hippocampal regions, and dividing these regions into thirds based on their anteroposterior extent. ROIs from the primary visual cortex in each hemisphere were taken from the Jülich atlas in FSL (44,45) and resampled to 2.5 mm. We selected the most medial, posterior portion of the primary visual cortex to capture cortical regions corresponding to foveal vision with an ROI of similar size to our MTL ROIs.
Representational Similarity Analysis. For each pair of images presented to the subjects, we calculated the geodesic distance in meters between the two GPS coordinates and the difference in time in seconds. Geodesic distance was calculated using the GeoPy Python package. Image pairs with spatial distances less than 100 m were excluded because these distances are below the reliability of the GPS radios in these smartphones. Image pairs with temporal distances below 15.6 h were excluded based on prior work because of a discontinuity in the spatiotemporal distribution of image pairs (19) (Fig. S2B). The discontinuity between 14-and 16-h results from subjects taking off their cameras to sleep. This gap is propagated through the rest of the results as a relative lack of image pairs that are multiples of ∼15 h apart. In prior work, we analyzed the structure of lifelogged images and demonstrated that image pairs taken from identical spatiotemporal locations occupied a lower dimensional manifold than those image pairs taken from separate spatiotemporal locations (19). By removing image pairs separated by less than 100 m and 15.6 h, we reduced the possibility that the images themselves would give rise to the present results as a consequence of within-and between-episode image properties. Some subjects spent time out of town during the period of data collection, resulting in a small portion of image pairs with spatial distances greater than 30 km; these image pairs were also excluded. Images that were blurry or contained reflections of the subjects were also excluded.
In our figures and description of our analysis, spatial distance is referred to as space and temporal distance is referred to as time. In each ROI, neural distance for each image pair was calculated as 1 minus the correlation between the voxel-level single-trial betas for the trials corresponding to those image pairs. Neural distances were z-scored within subjects. In each ROI in each subject, we then ran a GLM with the following model: neural distance = α + β time log 10 ðtimeÞ+ β space log 10 ðspaceÞ + β time:space log 10 ðtimeÞ*log 10 ðspaceÞ + β scanner log 10 ðscannertimeÞ + e.
We used the log of time and space based on previous literature that has shown a power-law relationship for neural representations (20). The scanner time was the number of seconds between presentation of the images during the fMRI experiment. In each ROI, we performed a t test on the betas from the subject-level GLMs to determine if they were significantly different from zero across subjects. We used nonparametric permutation to test for significance (21) because the pairwise nature of the distances in our analysis violated the assumption of independent samples. Neural data were permuted with respect to behavioral data within subjects. This process was repeated for 10,000 permutations of the neural data. The significance of t values was determined as the ratio of the number of permutations with higher absolute value of t values in that ROI for that term divided by the total number of permutations. We then applied the Bonferroni correction to control for the number of ROIs tested (eight MTL ROIs plus two bilateral primary visual cortex ROIs for a total of 10). To visualize the relationship between each term and the neural distance, we isolated the portion of the neural distance that was associated with changes in that term by subtracting the effects of the other terms as estimated on a subject level from the neural distance for each subject. For example, in the graphs for the relationship between space and neural distance, the neural distance has the estimates for the intercept, time, interaction of time and space, and time between presentations in the scanner subtracted out. To compare the significance of different regions for each term, we used the difference between t values between the two regions for that term in each of the 10,000 permutations as the null distribution to generate P values.