Pledge Agreement

EXHIBIT 10.4

PLEDGE AGREEMENT

(Stock) THIS PLEDGE AGREEMENT (" this Agreement" ) dated as of April 3, 2006, is between ALABAMA NATIONAL BANCORPORATION , a Delaware corporation, as pledgor (the " Pledgor" ) and AMSOUTH BANK , an Alabama banking corporation, as pledgee and secured party (the " Lender" ).

Recitals

A. The Pledgor is the holder, beneficially and of record, of certain shares of the outstanding capital stock of First American Bank, an Alabama banking corporation (the " Corporation" ), as more particularly described on Exhibit A attached hereto and made a part hereof (the " Stock" ). B. Capitalized terms used in these Recitals have the meanings defined for them above or in Section 1.2. The Pledgor has requested that the Lender extend a revolving loan (the " Revolving Loan" ) to the Pledgor as to be evidenced by that certain Revolving Note dated of even date herewith, executed by the Pledgor in favor of the Lender. To secure the Obligations, and to induce the Lender to extend the Revolving Loan to the Pledgor, the Pledgor has agreed to execute and deliver this Agreement to the Lender.

Agreement

NOW, THEREFORE , in consideration of the foregoing Recitals, and to induce the Lender to extend the Revolving Loan to the Pledgor, the Pledgor agrees with the Lender as follows:

ARTICLE 1 Rules of Construction and Definitions

SECTION 1.1 Rules of Construction .

For the purposes of this Agreement, except as otherwise expressly provided or unless the context otherwise requires:

(a) Words of masculine, feminine or neuter gender include the correlative words of other genders. Singular revolving include the plural as well as the singular, and vice versa.

(b) All references herein to designated " Articles," " Sections" and other subdivisions or to lettered Exhibits are to the designated Articles, Sections and subdivisions hereof and the Exhibits annexed hereto unless expressly otherwise designated in context. All Article, Section, other subdivision and Exhibit captions herein are used for reference only and do not limit or describe the scope or intent of, or in any way affect, this Agreement.

(c) The terms " include," " including," and similar terms shall be construed as if followed by the phrase " without being limited to."

(d) The terms " herein," " hereof" and " hereunder" and other words of similar import refer to this Agreement as a whole and not to any particular Article, Section, other subdivision or Exhibit.

(e) All Recitals set forth in, and all Exhibits to, this Agreement are hereby incorporated in this Agreement by reference.

(f) No inference in favor of or against any party shall be drawn from the fact that such party or such party' s counsel has drafted any portion hereof. (g) All references in this Agreement to a separate instrument are to such separate instrument as the same may be amended or supplemented from time to time pursuant to the applicable provisions thereof.

SECTION 1.2 Definitions . The following terms are defined as follows:

(a) Unless otherwise defined herein, terms used in this Agreement that are defined in Article 9 of the Alabama Uniform Commercial Code (the " UCC" ) have the meanings defined for them therein. (b) Additional Stock is defined in Section 2.2.

(c) Business Day means any day, excluding Saturday and Sunday, on which the Lender' s main office in Birmingham, Alabama, is open to the public for carrying on substantially all of its banking business.

(d) Default Rate means a rate of interest equal to four percentage points (four hundred basis points) in excess of the highest interest rate that would otherwise be payable on the principal amount of the Revolving Loan under the Revolving Note from time to time in the absence of the existence of a default, or the maximum rate permitted by law, whichever is less.

(e) Event of Default is defined in Section 4.1. An Event of Default " exists" if the same has occurred and is continuing.

(f) Governmental Authority means any national, state, county, municipal or other government, domestic or foreign, and any agency, authority, department, commission, bureau, board, court or other instrumentality thereof.

(g) Lien means any mortgage, pledge, assignment, charge, encumbrance, lien, security title, security interest or other preferential arrangement.

(h) Obligations means (1) the payment of all amounts now or hereafter becoming due and payable under the Revolving Note, including the principal amount of the Revolving

2

Loan, all interest thereon (including interest that, but for the filing of a petition in bankruptcy, would accrue on any such principal) and all other fees, charges and costs (including attorneys' fees and disbursements) payable in connection therewith; (2) the observance and performance by the Pledgor of all of the provisions of the Revolving Note; (3) the payment of all sums advanced or paid by the Lender in exercising any of its rights, powers or remedies under the Revolving Note, and all interest (including post-bankruptcy petition interest, as aforesaid) on such sums provided for herein or therein; (4) the payment of all amounts now or hereafter becoming due and payable under any agreement between the Pledgor and the Lender now existing or hereafter entered into, which provides for an interest rate or commodity swap, cap, floor, collar, forward foreign exchange transaction, currency swap, cross-currency rate swap, currency option, or any combination of, or option with respect to, these or similar transactions, for the purpose of hedging such Pledgor' s exposure to fluctuations in interest rates, currency valuations or commodity prices, and (5) all renewals, extensions, modifications and amendments of any of the foregoing, whether or not any renewal, extension, modification or amendment agreement is executed in connection therewith.

(i) Obligors means all " obligors" as defined in Article 9 of the UCC, the Pledgor and each other person, if any, executing any Security Document as a grantor, and any other maker, endorser, surety, guarantor or other person now or hereafter liable for the payment or performance, in whole or in part, of any of the Obligations.

(j) Permitted Encumbrances means any Liens and other matters affecting title to the Property that are described in Exhibit C .

(k) Person (whether or not capitalized) includes natural persons, sole proprietorships, corporations, trusts, unincorporated organizations, associations, companies, institutions, entities, joint ventures, partnerships, limited liability companies and Governmental Authorities. (l) Pledged Stock is defined in Section 2.2.

(m) Property is defined in Section 2.2.

(n) Security Documents means this Agreement and all documents that now or hereafter grant or purport to grant to the Lender any guaranty, collateral or other security for any of the Obligations.

ARTICLE 2

Security Agreement SECTION 2.1 Pledge of Stock .

As security for the Obligations, the Pledgor hereby grants to the Lender security title to and a continuing security interest in, a lien upon and assigns, transfers, conveys, pledges and hypothecates to the Lender, all of the Pledgor' s right, title and interest in and to the Stock and all proceeds and supporting obligations thereof, and, as applicable, the Pledgor hereby delivers to the Lender the stock certificate(s) evidencing the Stock, as described more fully in Exhibit A , together with separate assignments thereof, to be held by the Lender upon the terms and conditions set forth in this Agreement.


3

SECTION 2.2 Pledge of Additional Stock . If the Pledgor shall acquire by exchange or replacement any additional shares of capital stock, of whatever class or description (" Additional Stock" ) at any time after the date hereof, the Pledgor hereby grants to the Lender a security interest in, and assigns, transfers, conveys, pledges and hypothecates to the Lender, all of the Pledgor' s right, title and interest in and to the Additional Stock and such certificates, and immediately upon receipt thereof the Pledgor shall pledge and deposit the Additional Stock with the Lender and shall deliver to the Lender certificates therefor registered in the name of the Pledgor, together with executed separate assignments thereof, to be held by the Lender under this Agreement. The Stock, the Additional Stock, and any stock or other securities issued in exchange therefor or replacement thereof, are hereinafter together called the " Pledged Stock," and the Pledged Stock and all proceeds thereof and all other securities and moneys received and at the time held by the Lender hereunder are hereinafter together called the " Property," all of which shall be subject to the Liens granted to the Lender under this Agreement.

SECTION 2.3 Dividends and Other Distributions .

Unless an Event of Default exists, all cash dividends paid on the Pledged Stock shall be paid to the Pledgor, except that all cash dividends payable on the Pledged Stock that are determined by the Lender to represent in whole or in part an extraordinary, liquidating or other distribution in return of capital shall be paid to the Lender and retained by it as Property. The Lender shall also be entitled to receive directly and to retain as Property:

(a) all stock and other securities or property (other than cash) paid or distributed with respect to the Pledged Stock by way of dividend;

(b) all stock and other securities or property (including cash) paid or distributed with respect to the Pledged Stock by way of stock-split, spin-off, split-up, reclassification, combination of shares or similar or other corporate rearrangement; and (c) all stock and other securities or property (including cash) that may be paid or distributed with respect to the Pledged Stock by reason of any consolidation, merger, exchange of stock, conveyance of assets, liquidation or similar corporate reorganization.

SECTION 2.4 Voting While No Event of Default .

Unless an Event of Default exists, the Pledgor shall have the right to vote any and all shares of the Pledged Stock and to give consents, waivers and ratifications with respect to the Property and otherwise act with respect thereto. All such rights of the Pledgor to vote and to give consents, waivers and ratifications shall cease if an Event of Default exists.


4

ARTICLE 3

Representations, Warranties and Covenants

SECTION 3.1 Representations and Warranties .

The Pledgor represents and warrants to the Lender that (a) subject to Permitted Encumbrances, the Pledgor has the power to transfer or is the holder of record and sole beneficial owner of the Stock (which is fully issued and non-assessable), free of Liens and adverse claims of any kind, except Permitted Encumbrances; (b) the Pledgor has a good right to grant to the Lender the Liens in the Stock purported to be granted under this Agreement; (c) there are no outstanding subscriptions, options, rights, warrants, calls, commitments or agreements of any kind to acquire or transfer any of the Stock or to issue any additional shares of the capital stock, and there are no securities in existence that are convertible into any shares of such capital stock; (d) to the best of the Pledgor' s knowledge, no consent, authorization or other action by, and no notice to or filing with, any other person (including any stockholder, partner or creditor of the Pledgor and any Governmental Authority) is required for (1) the execution and delivery of this Agreement by the Pledgor, (2) the granting to the Lender of the Liens on the Property under this Agreement, or (3) the exercise by the Lender of the rights, powers and remedies granted to it under this Agreement, except as may be required in connection with any disposition by the Lender of the Property under laws affecting the offering and sale of securities generally; and (e) the location (including addresses, if applicable) of (1) each of the Pledgor' s places of business, (2) the Pledgor' s chief executive office, and (3) the Pledgor' s state of incorporation or registration (if the Pledgor was created by such state filing), are correctly and completely set forth on Exhibit D . The Pledgor' s legal name is as set forth in the first paragraph to this Agreement. No change has occurred in any of the foregoing in the five years immediately preceding the execution of this Agreement. SECTION 3.2 Encumbrances and Dispositions .

The Pledgor shall not (a) encumber any of the Property, or permit any of the Property to be encumbered, with any kind of Lien, other than Permitted Encumbrances, (b) sell, transfer or otherwise dispose of, or grant any option or warrant with respect to, any of the Property, or (c) permit the Corporation to issue any additional shares of its capital stock (to the extent that the Pledgor has the ability to prevent such issuance). SECTION 3.3 Taxes and Assessments .

The Pledgor shall pay when due all taxes, assessments and other charges levied or assessed against any of the Property, and all other claims that are or may become Liens against any of the Property, except any that are Permitted Encumbrances; and should default be made in the payment of same, the Lender, at its option, may pay them.

SECTION 3.4 Filing Fees and Taxes .

The Pledgor agrees, to the extent permitted by law, to pay all recording and filing fees, revenue stamps, taxes and other expenses and charges payable in connection with the execution and delivery of the Revolving Note or this Agreement, and the recording, filing, satisfaction, continuation and release thereof.


5

SECTION 3.5 Control . The Pledgor hereby grants control of the Property to the Lender, and the Pledgor shall take all actions requested by the Lender that the Lender deems in its sole discretion advisable to further establish such control, including obtaining control agreements from the applicable holders of the Property.

SECTION 3.6 Authorization .

The Pledgor authorizes the Lender to perfect, preserve, continue, amend and maintain the Lender' s interest in the Property by whatever actions the Lender in its sole discretion deems appropriate under applicable law. The Pledgor shall assist and cooperate with the Lender in taking such actions and shall pay all costs and expenses incurred by the Lender in taking such actions. Such actions may include without limitation (1) the Lender' s obtaining control of the Property; (2) the Lender' s filing of financing statements describing the Property; or (3) the Lender' s taking possession of the Property.

SECTION 3.7 Further Assurances .

At the Pledgor' s cost and expense and upon request of the Lender, the Pledgor shall duly execute and deliver, or cause to be duly executed and delivered, to the Lender such further instruments and do and cause to be done such further acts as may be reasonably necessary or proper in the opinion of the Lender or its counsel to perfect, preserve and protect the validity of the Liens of the Lender in the Property and to carry out more effectively the provisions and purposes of this Agreement.

SECTION 3.8 Attorney-in-Fact .

The Pledgor hereby constitutes and appoints the Lender, or any other person whom the Lender may designate, as the Pledgor' s attorney-in-fact, at the Pledgor' s sole cost and expense, effective upon the existence of any Event of Default, with full authority in the place and stead of the Pledgor and in the name of the Pledgor or otherwise, from time to time in the Lender' s discretion to take any action (a) that the Pledgor has agreed, but has failed, to take under this Agreement, (b) that the Lender in its sole discretion deems necessary or advisable to maintain, preserve or protect the security intended to be afforded by this Agreement, or (c) that the Lender may deem necessary or advisable to accomplish the purposes of this Agreement and the Revolving Note. SECTION 3.9 Release .

The Pledgor shall not file a release, amendment, partial release, or termination statement with respect to any of the Property without the Lender' s prior written consent.


6

...