We study the problem of optimal portfolio construction when the log-prices follow a discrete-time cointegrated vector autoregressive model. We follow the classical Markowitz mean-variance optimization approach, and derive expressions for the optimal portfolio weight vector over a single decision interval, both for a finite-time horizon and in the limit of an infinite horizon. It is often stated in the literature that given assets whose price dynamics exhibit cointegration, the portfolio weights should be chosen from the space of cointegrating relations, resulting in what is commonly referred to as the beta portfolio. However, we show here that the optimal action in the mean-variance sense for a finite trading interval is to buy the portfolio with a component both in the beta direction and a component in the direction of expected change. Furthermore, we prove that the beta portfolio is optimal only in the limit of an infinite trading horizon. Additionally, we derive the conditions under which the optimal portfolio is proportional to the disequilibrium readjustment forces of the cointegration model. Our results rely on a careful eigenanalysis of the underlying state space model, in which we derive a closed form solution for the optimal Markowitz portfolio, which is well-behaved despite the nonstationarity of the underlying price dynamics. We demonstrate our results with evaluations using both simulated and historical data.