Automatic detection on orbit is an efficient way to filter useless data downloaded to the ground. However, detection on orbit is a challenging task due to limited computational resources on the satellite. In this paper, a context-aware and depthwise-based detection framework for remote sensing images is proposed which can be used on orbit. In the result of limited computational resources on the satellite, on-orbit object detection should detect with low memory cost and fast speed while ensuring the accuracy. To address the problem of small model in the process of feature extracting, a depthwise convolution is applied instead of typical convolution. In this light, a small deep neural network is built to run on orbit, using Single Shot Multibox Detector (SSD) as basic detection module. Motivated by its weak performance on remote sensing image owing to few pixel about target object, context information about target object is added to improve performance. To further investigate the context information influence, we add a balance factor to balance the context information and background noise it brings. Then an experiment on real remote sensing image dataset is conducted comparing our extended model with other current state-of-the-art detection models. Results show our extended model outperforms other models in accuracy and speed. Deploying the pretrained model on the Android Platform with only 60M memory cost confirms the feasibility to detect on orbit. This detection system is to be verified on the TZ-1 satellite which will be launched in the year of 2018.