I'm trying to make a forest for my new land mod, but at this point it doesn't look good. Does anyone have any tips on how to make a good looking forest?

Well, I consider myself to be quite experienced in this area, so maybe I can help.

First, what kind of forest are you trying to make? Pine or aspen? Is it snowy or warm? Dense, or sparse? Based on those things you'll want to pick 2 or 3 primary ground textures, leaving the other 4 or 5 texture slots for detailing and 'special' things. You can also then figure out what kinds of objects you should be using.

General tips?:

Don't overuse the Smooth (S) tool. You'll notice that your forest will look a lot better if it's a little more bumpy and abrupt. Of course you still want to avoid sharp edges and landspikes, as they hardly look natural or good.

Be a bit random. Plan out the rough shape of an area, then just throw things around and have fun. You'll get a random result, which usually looks far more natural than if you had planned the area object for object.

Expect change. Don't plan an area then try to stick directly to that. Go with the flow. If you place something and like the way it looks, but it conflicts with something else you wanted to place (That mountain is now blocking where you were going to put the stream), build the other object around the first. Try to make it work. The result will be a more complex random looking feature, something that nature is full of.

Those are the things that immediately jump to mind. If you provide more detail on what you're going for, I can provide some more tips.

My forest actually has two sections, both are moss and mud covered so I am primarily using those terrains. The first section is a medium thick forest using thinner trees. I am using modded trees (thanks GKB) as well as trees original to the game. What I want to make is essentially a forest that looks like a Japanese forest I can show you the images I got my idea from.

http://www.gettyimages.co.uk/detail/photo/japan-kyoto-japanese-forest-garden-elevated-high-res-stock-photography/200213565-001

http://japanstocks.deviantart.com/art/Japanese-Forest-79233411

Also thanks

Ah, well. Keep in mind if you expect to get that exactly level of detail and beauty, you'll probably be disappointed. But, instead, if you simply aim to make something similiar you should be okay.

Just be rather random with how you place things. Make the ground uneven, like I said, dont oversmooth. It may be hard to do without additional new resources, like smaller creeks. The Skyrim ones look rough and mountainous, which wont exactly fit your terrain. But you can always get creative with them.

If you want further help, all I can think of is you taking a screenshot and I can maybe give some pointers off that.

How do you take screen shots while in game? Sorry I'm new to this and still learning.

I don't expect it to be that detailed or good. Right now I think the trees I'm using are either too colorful or too big

If you hit F12 steam takes one, then close the game and say Upload. You can then post a link directly to that screenshot.

Here it is. Right now I'm just working on getting the trees to look right, then I'm adding the ground cover.
Now that I take a closer look it does seem a bit to smooth

http://steamcommunity.com/sharedfiles/filedetails/?id=98081676

http://steamcommunity.com/profiles/76561198057465753/screenshots/?appid=72850

my stream
http://steamcommunity.com/profiles/76561198057465753/screenshots/?appid=72850