This is false as shown by the following digraph.
From $x$ there is an edge to $v_p$, from $v_p$ there is a cycle of length $p$ to itself, and from $v_p$ there are $p-1$ different paths to $y$, of lengths $1,2,\ldots,p-1$.
Then using these $\Theta(p^2)$ vertices we can get from $x$ to $y$ through $v_p$ along a walk of length $\ell$ for any $\ell\not\equiv 1 \pmod p$.
If you repeat the above construction for every prime $p\le N$, and also add the edge $xy$, then the shortest missing length is $1+\prod_{p\le N \text{ prime }} p$, which is much bigger than $\sum_{p\le N \text{ prime }} \Theta(p^2)$.
It would be a nice problem to determine exactly up to what length you need walks to get walks of any length.
Update. Apparently, Aleksei has solved essentially the same question five years ago, with an asymptotically optimal bound.