Trajectory data is essential for understanding the driver-vehicle-road interaction which is crucial for safety and operational assessment. The video image-processing technique is useful to collect such data on the entire traffic stream. Existing techniques assume that the road is planar which limits the application of the video-based traffic data collection to the plain terrains. With the objective to collect trajectory data from mountainous terrain, the present study proposes an approach to convert the pixel coordinates into real-world coordinates. The hypothesis was that a non-planar scene could be converted into a sequence of piece-wise planar scenes with a separate projection matrix for each planar region. The analysis shows that the proposed method could effectively transform the pixel coordinates into the real-world coordinates. Comparison of the estimated and observed speed/path profiles indicate the adequacy of the proposed method in collecting video-based trajectory data from mountainous terrain. © 2022 Informa UK Limited, trading as Taylor & Francis Group.