For your first question, you can indeed utilize the Add Form Fields API to set up a Field Group.
Interestingly, it's not clearly outlined in the English documentation, but the type you're looking for is GROUP for properties.{fieldcode}.type.
Just a quick sidebar: though not related to your query, Group Selection is defined as GROUP_SELECT.
Now, after you've successfully placed a group field in a form, you would have to engage the Update Form Field API to place fields within that group field.
For your second query, you can use the Get Form Field API to figure out whether the field has the "Prohibit duplicate values" option enabled.
This is identified through the properties.{fieldcode}.unique parameter.
If it returns true, duplicate values are not allowed, but if it returns false, you are in the clear.
Remember, you can also find the related app in the properties.{fieldcode}.lookup.relatedApp parameter.