One of the main problems in all-optical packet switched networks is the buffering of packets. A popular solution for buffering packets is to use a set of fiber delay lines attached to a switch. A priority queue is one of the most general buffering schemes that allows the packet with the highest priority to depart the buffer on a departure request and dropping of the lowest priority packet if a new packet arrives when the buffer is full. We present a recursive algorithm for constructing optical priority queues of polynomial size from a switch with linear number of inputs/outputs and fiber delay lines. The best known lower bound allows the construction of a priority queue of size 2^M using a switch of size M. However, the best known upper bound constructs a priority queue of size M³ using a switch of size M. We show that it is possible to construct a priority queue of size O(M^C) for a fixed constant c using a switch of size O(M).