Understanding human trajectories is key to many research areas such as physics, computer science and social sciences. Being able to learn behaviors with noninvasive sensors is important to analyzing the natural behaviors of humans. This problem has been widely studied in computer graphics, computer vision and machine learning. Existing approaches generally fall into model-based and model-free methods. Early model-based methods tended to be empirical or rule-based methods derived via the first-principles approach: summarizing observations into rules and deterministic systems based on fundamental assumptions on human motion. In such a perspective, social interactions can be modelled as forces in a particle system or an optimization problem, and individuals can be influenced by affective states. Later, data-driven model based methods were introduced, in which the model behavior is still dominated by the assumptions on the dynamics, e.g. a linear dynamical system, but retains sufficient flexibility so that the model can be adjusted to fit observations. More recently, model-free methods based on deep learning have also been explored, and these demonstrate surprising trajectory prediction capability. While empirical or rule-based methods possess good explainability, they are formed to model specific behaviors and it is hard to calibrate them on data. Data-driven model-based methods (e.g., statistical machine learning) improve the ability of data fitting but are restricted by the specific statistical models employed which have limited capacities to learn from large amounts of data. Finally, deep learning approaches excel at data fitting, but lack explainability and therefore have been mainly used for prediction rather than analysis and simulation. In this project, we explore a new class of models where deep learning is enhanced with inductive biases inspired by physics.