Official bonds of public officers shall not be discharged by a single recovery, but proceedings against the officer or his sureties or both may be had from time to time until the whole penalty is exhausted; and said bonds are joint and several whether or not so set forth.