Picking the reflection horizon is an important step in velocity inversion and seismic interpretation. Manual picking is time-consuming and no longer suitable for current large-scale seismic data processing. Automatic algorithms using different seismic attributes such as instantaneous phase or dip attributes have been proposed. However, the computed attributes are usually inaccurate near discontinuities. The waveforms in the horizontal direction often change dramatically, which makes it difficult to track a horizon using the similarity of attributes. In this paper, we propose a novel method for automatic horizon picking using multiple seismic attributes and the Markov decision process (MDP). For the design of the MDP model, the decision time and state are defined as the horizontal and vertical spatial position on a seismic image, respectively. The reward function is defined in multi-dimensional feature attribute space. Multiple attributes can highlight different aspects of a seismic image and therefore overcome the limitations of the single-attribute MDP through the cross-constraint of multiple attributes. The optimal decision is made by searching the largest state value function in the reward function space. By considering cumulative reward, the lateral continuity of a seismic image can be effectively considered, and the impacts of abnormal waveform changes or bad traces in local areas for automatic horizon picking can be effectively avoided. An effective implementation scheme is designed for picking multiple reflection horizons. The proposed method has been successfully tested on both synthetic and field data.