let’s not means: 1. You say no to an invitation to participate in a particular activity. 2. A way to calm someone down or stop. Three “Let’s Not” may be followed by “with”, + nominal construction. (in Community Dictionary, added by Snicket)